PC*T WORLD INTELLECTUAL PROPERTY ORGANIZATION ^aSBy 

^ **■ Internationa] Bureau lu * 

INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT) 



(51) International Patent Classification 6 : 
G06K 9/00 



Al 



(11) International Publication Number: 



WO 99/04361 



(43) International Publication Date; 28 January 1999 (28.01.99) 



(21) International Application Number: PCT/US98/14776 

(22) International Filing Date: 16 July 1998 (16.07.98) 



(30) Priority Data: 

60/052,902 



16 July 1997 (16.07.97) 



US 



(71) Applicant (for all designated States except US): DIVERSIFIED 

SCIENTIFIC, INC. [US/US]; Suite 381, 2800 Milan Court, 
Binningham, AL 35211 (US). 

(72) Inventors; and 

(75) Inventors/Applicants (for US only): GESTER, Thomas, E. 
[US/US]; 3020 Briarcliff Road, Mountain Brook, AL 35223 
(US). ROSENBLUM, William, M. [US/US]; 2690 Altadena 
Road, Birmingham, AL 35243 (US). CHRISTOPHER, 
Gayle, K. [US/US]; 788 Lake Wood Drive, Harpersville, 
AL 35078 (US). HAMRICK, David, T. [US/US]; 745 
Rifle Range Road, Glencoe, AL 35905 (US). DELUCAS, 
Lawrence, J. [US/US]; 2739 Altadena Road, Birmingham, 
AL 35243 (US). TILLOTSON, Brian [US/US]; 24231 146th 
Avenue SJB. f Kent, WA 98042 (US). 



(74) Agents: COGEN, Ellen, S. et aL; Gifford, Krass, Groh, 
Sprinkle, Patmorc, Anderson & Citkowski, P.C., Suite 
400, 280 N. Old Woodward Avenue, Binningham, MI 
48009-5394 (US). 



(81) Designated States: AL, AM, AT, AU, AZ, BA, BB, BG, BR, 
BY, CA, CH, CN, CU, CZ, DE, DK, EE, ES, FI, GB, GE, 
GH, GM, HR, HU, ID, EL, IS, JP f KB, KG, KP, KR, KZ, 
LC, LK, LR, LS, LT, LU, LV, MD, MG, MK, MN, MW, 
MX, NO, NZ, PL, FT, RO, RU, SD, SE, SG, SI, SK, SL, 
TJ, TM f TR, TT, UA, UG t US, UZ, VN, YU, ZW, AR1PO 
patent (GH, GM, KB, LS, MW, SD, SZ, UG, ZW), Eurasian 
patent (AM, AZ, BY, KG, KZ, MD, RU, TJ, TM), European 
patent (AT, BE, CH. CY, DE, DK, ES, FI, PR, GB, GR, 
IE, IT, LU, MC, NL, PT, SE), OAPI patent (BF, BJ, CF, 
CG, CI, CM, GA, GN, GW, ML, MR, NE, SN, TD, TG). 



Published 

With international search report. 
With amended claims. 



BEST AVAILABLE COPY 




KEYBOARD 



24 

n 



CPU 



(54) Title: METHOD FOR ACQUIRING, STORING AND ANALYZING CRYSTAL IMAGES 
(57) Abstract 

A system utilizing a digital computer 
for acquiring, storing and evaluating crystal 
images. The system includes a video cam- 
era (12) which produces a digital output sig- 
nal representative of a crystal specimen po- 
sitioned within its focal window (16). The 
digitized output from the camera (12) is 
then stored on data storage media (32) to- 
gether with other parameters inputted by a 
technician and relevant to the crystal spec- 
imen. Preferably, the digitized images are 
stored on removable media (32) while the 
parameters for different crystal specimens 
are maintained in a database (40) with in- 
dices to the digitized optical images on the 
other data storage media (32). Computer 
software is then utilized to identify not only 
the presence and number of crystals and the 
edges of the crystal specimens from the op- 
tical image, but to also rate the crystal spec- 
imens by various parameters, such as edge 
stralghtness, polygon formation, aspect ra- 
tio, surface clarity, crystal cracks and other 
defects or lack thereof, and other parame- 
ters relevant tothe quality of the crystals. 
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METHOD FOR ACQUIRING, STORING 
AND ANALYZING CRYSTAL IMAGES 

Background of the Invention 
L Field of the Invention 
5 The present invention relates to a computer controlled system which 

acquires, stores and analyzes crystal images and other parameters relevant to the 
crystals, or microscopic details of other specimens. 

II. Description of the Prior Art 
In many chemical, pharmaceutical and medical applications crystals, e.g. 
10 protein crystals, are grown in trays for subsequent evaluation by a lab technician 
or scientist (hereinafter collectively referred to as "technician"). In evaluating the 
crystals once grown, the lab technician examines the crystals under the microscope 
and then visually evaluates or rates individual crystals. The straightness of the 
crystal edges, size of the crystal, presence or absence of flaws in the crystal as well 
15 as other crystal parameters are used by the technician in his or her rating process. 
The technician may also maintain notes of other parameters, such as pH, crystal 
growing time, temperature, et cetera, which are relevant to the particular crystal. 

In some instances, the technician will take and maintain a photograph of 
the particular crystal under examination. The photograph is then stored along with 
20 the notes relevant to the particular crystals in the particular tray. 

This previously known system for evaluating and rating crystals is 
disadvantageous for a number of reasons. Most prominently, the crystal 
evaluation and rating system is labor intensive and, thus, not only slow and 
expensive in labor costs, but also tedious for the technician. 
25 A still further disadvantage of these previously known systems is that the 

photographic record used for the examined crystal is expensive in material costs 
from the photographic process. Furthermore, photographs by their very nature are 
easily damaged and also deteriorate over time. 

A still further disadvantage of the previously known method for evaluating 
30 and rating crystals is that no efficient means or system has been previously known 
for cross-referencing the various crystals and crystal parameters relative to each 
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other. Instead, the photographic picture as well as the other parameters relevant 
to the particular picture are simply maintained separately from the crystal 
specimens. 

Summary of the Pm^nt T nvf 

5 The present invention provides a system utilizing a digital computer which 

overcomes all of the above-mentioned disadvantages of acquiring, storing and 
evaluating crystals, such as protein crystals, or microscopic details of other 
specimens. 

In brief, the present invention utilizes a video camera which provides a 

10 digital output signal representative of an object, in this case crystals maintained 
within a tray, positioned within its focal window. Preferably, a central computer 
controls a movable stage to sequentially position trays in the focal window* The 
camera is operated under the control of a central computer which not only 
activates the initiation of the image acquisition by the camera, but also stores the 

15 output from the camera in digital form on data storage media. Preferably, the 
optical image from the video camera is stored on removable data storage media, 
such as a JAZZ™ drive, ZIP™ drive or CD ROM, etc.. 

In addition to storing the acquired optical image from the video camera, 
the technician, via a computer keyboard, mouse or other computer input means, 

20 also inputs data corresponding to parameters relevant to the particular crystal 
specimen under examination. Such parameters can include, for example, the pH, 
temperature, duration of crystal growth, et cetera for the particular crystal 
specimen. Additionally, the computer software preferably provides input template 
configurations to simplify the data input of the parameters by the technician and 

25 thus avoid or at least minimize the redundant input of information for different 
trays having similar parameters. 

The various stored parameters may also be stored with the optical image 
on the data storage media. However, more preferably, the parameters relevant to 
the crystal specimens are stored in a database on one data storage media, for 

30 example a hard drive, with an index or record pointer to the appropriate image 
stored on the removable drive with the optical images. In this fashion, a large 
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database of the various crystal parameters may be maintained and analyzed 
relative to each other with access to the optical images always available as 
required or desired. 

Following acquisition of the optical image as well as the other crystal 
parameters, the computer is programmed to analyze the optical image for the 
presence and count of protein crystals. As a part of the evaluation, the computer 
program identifies the edges of the crystal, filling in any gaps of the edge where 
necessary, and than analyzes the resulting data for its perimeter symmetry and 
roughness, straightness, crystal size, presence or absence of defects and center of 
gravity. The crystal rating is then stored in the data base. 

Preferably, the present invention utilizes a fast T-squared filer during its 
analysis of the optical image in order to grade the crystal. Alternatively, the 
present invention utilizes a 3x3 edge detection filter during its analysis of the 
optical image in order to identify the crystal edges, then the image is converted to 
a binary image with a threshold of approximately 40 on a scale of 0 to 255 to 
reduce image artifacts. The size of the crystal is determined via perimeter 
connectivity analysis. Objects with a small perimeter are excluded. The net 
resulting image is analyzed for its roughness which js a measure of the perimeter 
divided by the convex perimeter. This metric is used to isolate the crystal from 
the drop boundary and other artifacts. The center of gravity is calculated on the 
remaining data to pinpoint the crystals. 

Alternatively, other methods may also be used to grade the crystal based 
upon metrics such as edge straightness, aspect ratio, surface clarity, polygon 
formation, color etc. These methods include the use of traditional spatial filters 
such as highpass, lowpass, Butterworth, homomorphic, Sobel, Laplacian, etc. 
Probabilistic restoration such as least mean square (Wiener) filters, fast T-squared 
filters, spatial transformations, frequency transformations, etc. can be used. Edge 
linking and boundary detection using Hough Transforms, "Line-filler" filters, 
thresholding, etc. can be used. Representation and description using Fourier 
descriptors, topological descriptors, texture descriptors, statistical descriptors, 
moments, mathematical morphological descriptors, etc. can be used. Recognition 
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using minimum distance classifiers, correlation classifiers, statistical classifiers, 
Bayesian classifiers, neural networks, genetic algorithms, etc, can be used. 

Brief Dftsmprinti of thf; T)rawinc 
A better understanding of the present invention will be had upon reference 
to the following detailed description when read in conjunction with the 
accompanying drawing, wherein like reference characters refer to like parts 
throughout the several views, and in which: 

FIG. 1 is a block diagrammatic view illustrating a preferred embodiment 
of the present invention; 

FIG. 2 is an exemplary image of a crystal; and 

FIGS. 3 through 6 are flow charts illustrating the operation of the system 
of the present invention, 7 

Detailed Description of a Preferred 
Embodiment nf th* Pre^ t Tnvp.ntin n 

With reference first to FIG. 1, a block diagrammatic view of a preferred 
embodiment of the system of the present invention is there shown and comprises 
a video camera 12 which, upon activation, generates a signal on its output line 10 
representative of the optical image in its focal window 16. A tray 18 containing 
the crystals under examination is positioned within the focal window 16. 

Although the tray 1 8 can be positioned manually within the focal window 
16 of the camera 12, in the preferred embodiment of the invention, preferably a 
plurality of trays 18, e.g. two trays, are positioned on a movable stage at a time 
and each tray contains twenty-four to ninety-six wells while a motorized 
mechanism 20 is used to control the position of the stage 22. A central processing 
unit or computer 24, under program control, then controls the activation of the 
motorized mechanism 20 by generating appropriate signals on line 25 and thus 
controls the position of the stage 22. In doing so, the motorized mechanism 20 
automatically sequentially positions the individual wells 18 within the focal 
window 16 of the camera 12 so that an optical image of each well 18 is acquired. 
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In one embodiment, the computer 24 can control the ability of the system to focus, 
as well as zoom in/out on the specimen, and then store this information. 

Once the motorized mechanism 20 has positioned a desired well 1 8 in the 
focal window 16, the central processing unit 24 generates an output signal on its 
5 output line 26 to the camera 12 in order to activate the camera 12 to produce 
signals on its output line 14 representative of the crystal image. These output 
signals are fed as input signals to the central processing unit 24 either directly via 
a port on the central processing unit 24 or via an appropriate input device 28, such 
as an appropriate video card. 

10 In order to ensure even illumination of the crystal specimen during the 

image acquisition by the camera 12, preferably a fiber optic lighting system 30 is 
utilized as a back light under the tray 16. In one embodiment, the computer 24 
can control the intensity of the lighting system 30, as well as the presence of light 
Additionally, the computer 24 can also control polarizing means, such as the angle 

1 5 of polarization. Other illumination means, however, may alternatively be used. 

Following acquisition of the output signal from the video camera 12, the 
central processing unit 24 stores the image on data storage media using a storage 
device 32. Since a digital representation of an optical image typically consumes 
a relatively great amount of memory, e.g. one megabyte, the data storage device 

20 32 is preferably a removable data storage media such as a JAZZ™ drive, ZIP™ 
drive or CD ROM, etc. Other types of media, including other types of removable 
data storage media, may alternatively be used. 

During the data acquisition of the image by the central processing unit 24, 
the technician also utilizes a keyboard 34 or other data input equipment together 

25 with a video monitor 36 to input data pertinent to the particular crystal specimen. 
Such data input can include, for example, the pH of the specimen, temperature of 
the specimen, protein type of the specimen, detergents, additives, preservatives, 
reservoir buffer and associated variables of concentration, pH and volume, notes, 
score, drop descriptor, etc. Furthermore, in order to facilitate the entry of these 

30 additional parameters by the technician, the central processing unit 24 preferably 
utilizes software templates displayed on the video monitor 36 to eliminate, or at 
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least minimize, the entry of redundant or repetitive information by the technician. 
Examples of such are included in Appendix A, incorporated herein. 

• The various parameters for each crystal specimen are also stored on data 
storage media, such as a hard drive 40, in a database and preferably in a relational 
database. Since the various parameters pertinent to the crystal specimens consume 
considerably less storage space than the optical images themselves, preferably the 
parameters are stored on the hard drive 40 separate from the optical images stored 
by the data storage device 32. In doing so, the parameters stored on the hard drive 
40 would include an index or pointer to the optical images stored by the data 
storage media unit 32 so that the various crystal parameters stored in the database 
may be easily correlated whenever desired to the optical images. 

The storage of the optical images of the crystals on data storage media as 
well as the storage of the crystal parameters in a database, such as a relational 
database, provide substantial advantages over the previously known photography 
and handwritten notes currently utilized by technicians. Perhaps most 
importantly, the use of a relational database allows the data to be analyzed, sorted, 
correlated and otherwise manipulated under software control by the central 
processing unit 24. The storage of the optical images on data storage media also 
provides a permanent record of the images without the previously known 
degradation and possible damage of photographs. 

With reference now to FIG. 2, an exemplary image of a crystal stored by 
the central processing unit on data storage media is there shown. The crystal 150 
includes a plurality of edges 152 having a certain degree of straightness. 
Additionally, the edges 152 of the image of the crystal 150 may include gaps 154 
due to lighting deficiencies during the image acquisition, deficiencies of the 
camera 12, or for other reasons. 

With reference now to FIG. 3, a flow chart illustrating the operation of the 
present invention is there shown. At step 50, the program is initialized and the 
central processing unit 24 generates signals on its output line 25 to the motorized 
mechanism 20 (FIG. 1) in order to move the stage 22 to position the first crystal 
tray well 18 within the focal window 16 of the video camera 12. Step 50 then 
branches to step 52. At step 52, the central processing unit 24 generates an output 
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signal on line 26 to activate the video camera 12 to acquire the optical image and 
to input the optical image from the camera output line 10 of the camera 12 to the 
central processing unit 24. Step 52 then branches to step 54. 

At step 54, the central processing unit 24 stores the optical image via the 
5 data storage device 32 (FIG. 1). Optionally, appropriate data compression 
software may be utilized to minimize the storage required by the optical image of 
the crystal storage. Likewise, optionally digital filtering algorithms may be also 
utilized to remove extraneous noise and otherwise enhance the digital image prior 
to storage on the data storage media by the storage device 32 or 40. 

10 Concurrently with storing the optical image of the crystal on data storage 

media, the technician inputs various relevant parameters, such as pH, temperature, 
crystal growth duration, protein type, et cetera pertinent to the crystal under 
examination via the keyboard 34 or other input device. Preferably, the central 
processing unit 24 under software control provides appropriate templates to the 

1 5 technician replicating various parameters common to the various specimens in the 
trays 18 on the stage 22. In doing so, the entry of redundant or duplicative 
information by the technician between crystal specimens is minimized which 
minimizes not only the labor required from the lab technician, but also the entry 
of errors through human technician error, 

20 The various crystal parameters entered by the technician are also stored at 

step 54 by the central processing unit 24 on the data storage media 32 or 40 (FIG. 
1). Stored concurrently with the crystal parameters on the data storage media 32 
or 40 is an index or pointer which correlates the stored crystal parameters to the 
stored image on data storage media by the data storage device 32. 

25 Step 54 then branches to step 56 whereupon the central processing unit 24 

generates output signals on its output line 25 to activate the motorized staged 
mechanism 20 to move the stage 22 to position the next well in the focal window 
16 of the camera 12. Step 56 then branches to step 52 where steps 52 and 54 are 
reiteratively executed until an optical image and crystal parameters for each of the 

30 wells 1 8 on the stage 22 have been acquired and stored as previously described. 
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With reference now to FIGS. 4 and 5, following the acquisition and storage 
of the images and crystal parameters, step 58 is then executed to analyze the stored 
image. Step 58 is illustrated in greater detail in FIG. 5 and FIG. 6. 

With reference then to FIG. 5, during the analysis of the stored image, the 
5 program at step 60 executes a fast T-squared filter in order to locate and identify 
the edges of the crystal image. The fast T-squared filter is essentially a local 
digital filter which looks for a line defined as a set of co-linear pixels having a 
mean value different from the mean value of adjacent pixels on both sides of the 
line. Preferably, the fast T-squared filter at step 60 computes the two sample T- 
1 0 statistic with a null value of 0. 

Preferably, the T-squared filter utilizes integer mathematics to minimize 
computing time required by the central processing unit. The T-squared filter at 
step 60 is also preferably a variable width T-filter which identifies lines having 
variable widths, e.g, a width varying in size from one pixel to five pixels. Step 60 
1 5 then branches to step 62. 

At step 62 a low pass filter is applied to the image to remove extraneous 
noise. Step 62 then branches to step 64. 

At step 64, the program "fills" digitally any gaps, such as the gap 54 in 
FIG. 2, which may be present in the digitized image. Thus, following step 64, the 
20 program has identified not only the lines of the crystal, but also eliminated 
extraneous noise in the crystal image as well as completed any missing segments 
of the crystal edges due to deficiency of the camera, lighting or the like. Step 64 
. then branches to step 66 (FIG. 4). 

At step 66, the program "scores" or evaluates the crystal images and then 
25 stores the score or evaluation as one of the crystal parameters on the storage 
device 40. Such scoring includes examination and evaluation of the straightness 
of the crystal edges, the presence or absence of defects in the crystal, fractures in 
the crystal and the like. 

With reference then to FIG. 6, during the analysis of the stored image, the 
30 computer is programmed to analyze the optical image for the presence and count 
of protein crystals. As a part of the evaluation, the computer program identifies 
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the edges of the crystal and than analyzes the resulting data for its perimeter 
symmetry and roughness and center of gravity. 

Preferably, the present invention utilizes a 3x3 edge detection filter during 
its analysis of the optical image in order to identify the crystal edges, then the 
5 . image is converted to a binary image with a threshold of approximately 40 on a 
scale of 0 to 255 to reduce image artifacts. The size of the crystal is determined 
via perimeter connectivity analysis. Objects with a small perimeter are excluded. 
The net resulting image is analyzed for its roughness which is a measure of the 
perimeter divided by the convex perimeter. This metric is used to isolate the 
10 crystal from the drop boundary and other artifacts. The center of gravity is 
calculated on the remaining data to pinpoint the crystals. 

From the foregoing, it can be seen that the present invention provides a 
system for the automatic acquisition, storage and evaluation of crystal specimens 
under computer control. The use of software templates by the operator to 
1 5 minimize the entry of duplicative or repetitive information further enhances not 
only the reliability of the data inputted into the system, but also minimizes 
technician labor. 

A still further advantage of the present invention is the storage of the 
crystal parameters in a relational database which enables the manipulation and 
20 evaluation (i.e., experimental history, archiving, trend analysis, etc.) of the crystal 
parameters under computer control utilizing search and 3-D plotting capabilities. 
The storage of the crystal images in removable data storage media separate from 
the storage of the crystal parameters also ensures that a large number of crystal 
parameters can be stored in the relational database with access to the crystal 
25 images always available. 

Having described our invention, however, many modifications thereto will 
be come apparent to those skilled in the art to which it pertains without deviation 
from the spirit of the invention as defined by the scope of the appended claims. 

We claim: 
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Claims 

1 1 . A system for utilizing a digital computer to evaluate microscopic 

2 details of specimens, e.g. crystals, comprising: 

3 a camera which generates an output signal representative of an image 

4 positioned in a focal plane of the camera, 

5 means for automatically positioning a specimen in said focal plane, 

6 means for acquiring and storing said output signal from said camera, 

7 means for evaluating said stored image from said camera and for 

8 generating a result signal representative thereof, 

9 means for outputting said result signal. 

1 2. The invention as defined in claim 1 wherein said evaluating means 

2 comprises means for determining presence of crystals* 

1 3 . The invention as defined in claim 2 wherein said evaluating means 

2 further comprises means for determining the count of said crystals. 

1 4. The invention as defined in claim 1 and comprising means for 

2 generating three-dimensional surface plots of experimental data within a database. 

1 5. The invention as defined in claim 1 and comprising means for 

2 automatically controlling zoom and focus. 
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AMENDED CLAIMS 
[received by the International Bureau on 11 December 1998 (1 1.12.98); 
original claims 1-5 amended; new claims 6-14 added (2 pages)] 



1 LA system for utilizing a digital computer to evaluate microscopic 

2 details of specimens, e.g. crystals, comprising: 

3 a camera which generates an output signal representative of an image 

4 positioned in a focal plane of the camera, 

5 a tray which positions a specimen in said focal plane, 

6 a computer processing unit having a persistent storage device, which 

7 computer processing unit acquires said output signal from said camera, 

8 said computer processing unit being programmed to evaluate said stored 

9 image from said camera and for generating a result signal representative thereof, 

10 said computer processing unit being programmed to store said result 

1 1 signals in said persistent storage device. 

1 2. The invention as defined in claim 1 wherein each said specimen 

2 comprises at least one crystal. 

1 3. The invention as defined in claim 2 wherein said computer 

2 processing unit is programmed to count said crystals. 

1 4. The invention as defined in claim 2 wherein said computer 

2 processing unit is programmed to generate three-dimensional surface plots of 

3 crystals within a database. 

1 5. The invention as defined in claim 1 and comprising a movable 

2 stage for automatically positioning said specimens in said focal plane. 

1 6. The invention as defined in claim 2 and comprising a T-squared 

2 filter to identify said crystals in each said specimen. 

1 7. The invention as defined in claim 6 where said T-squared filter 

2 comprises a software program executed by said computer processing unit. 
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1 8. The invention as defined in claim 5 wherein said computer 

2 processing unit generates output signals to control the movement of said movable 

3 stage. 

1 9. The invention as defined in claim 4 wherein data relating to said 

2 crystal(s) is stored in the database. 

1 10. The invention as defined in claim 1 and comprising a light source 

2 directed to said specimen. 

1 11. The invention as defined in claim 10 and comprising an optical 

2 fiber extending between said light source and said specimen. 

1 12. The invention as defined in claim 2 and comprising a computer 

2 algorithm executed by said computer processing unit for rating said crystals with 

3 respect to predetermined standards. 

1 13. The invention as defined in claim 2 and comprising a computer 

2 algorithm executed by said computer processing unit for simulating edges of 

3 crystals missing in said image generated by said camera. 

i 

1 14. The invention as defined in claim 2 wherein said computer 

2 processing unit is programmed to determine crystal size by determination of the 

3 length of the perimeter of said crystals. 
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